Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 1: Dataset Exploration

Visualize the German Traffic Signs Dataset. This is open ended, some suggestions include: plotting traffic signs images, plotting the count of each sign, etc. Be creative!

The pickled data is a dictionary with 4 key/value pairs:

  • features -> the images pixel values, (width, height, channels)
  • labels -> the label of the traffic sign
  • sizes -> the original width and height of the image, (width, height)
  • coords -> coordinates of a bounding box around the sign in the image, (x1, y1, x2, y2)
In [1]:
# Load pickled data
import numpy as np
import pickle

import skimage.transform
from sklearn.preprocessing import LabelBinarizer
%pylab inline

# TODO: fill this in based on where you saved the training and testing data
training_file = './../Downloads/train.p'
testing_file = './../Downloads/test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']
Populating the interactive namespace from numpy and matplotlib
In [2]:
n_train = X_train.shape[0]
n_test = X_test.shape[0]
image_shape = X_train[2].shape[0:]
n_classes = len(set(y_train))

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
print(X_train.shape)
Number of training examples = 39209
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43
(39209, 32, 32, 3)
In [3]:
import pandas as pd
df = pd.read_csv("signnames.csv")

def draw(X, y):
    for j in range(3):
        figure(figsize=(20,10))
        for i in range(5):
            subplot(151+i)        
            title(df.iloc[y[i+j*5]]['SignName'])
            imshow(X[i+j*5])
        show()

print("Look at Training")
draw(X_train, y_train)
print("Look at Testing")
draw(X_test, y_test)
Look at Training
Look at Testing

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

There are various aspects to consider when thinking about this problem:

  • Your model can be derived from a deep feedforward net or a deep convolutional network.
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

In [4]:
from skimage.exposure import equalize_hist
from skimage.exposure import equalize_adapthist,adjust_log

X_test = equalize_hist(X_test)
X_train = equalize_hist(X_train) 
# Alternative, but slightly worse results better for low lightning
#X_train = [equalize_adapthist(data,kernel_size=32) for data in X_train]
#X_test = [equalize_adapthist(data,kernel_size=32) for data in X_test]

print("Look at Training")
draw(X_train, y_train)
print("Look at Testing")
draw(X_test, y_test)
Look at Training
Look at Testing

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [5]:
binarizer = LabelBinarizer().fit(y_train)
y_train = binarizer.transform(y_train).astype(np.float32)
y_test = binarizer.transform(y_test).astype(np.float32)

from sklearn.model_selection import train_test_split
from sklearn.utils import shuffle
X_train, y_train = shuffle(X_train, y_train, random_state=42)
X_v, X_t, y_v, y_t = train_test_split(X_test, y_test, test_size=0.5, random_state=42)
In [6]:
print("Look at Training")
draw(X_train, np.argmax(y_train, axis=1))
print("Much better...")
Look at Training
Much better...

Question 1

Describe the techniques used to preprocess the data.

Answer:

I did no do that much preprocessing:

  • Histogram equalization to help the network a little bit on the very dark low contrast images
  • Shuffle the data (well that is obvious)
  • LabelBinarizer to get to the 43 class vector
  • Train test splitting.

Specifically I did not center the images values around 0 as reccommended sometimes (but not always). I tried both but got very similar results

In [7]:
# Additional Data and iterator functions
# Note: inspired by keras.preprocessing.image

import scipy.ndimage as ndi
import itertools
def apply_transform(x, transform_matrix):
    x = np.rollaxis(x, 2, 0)
    final_affine_matrix = transform_matrix[:2, :2]
    final_offset = transform_matrix[:2, 2]
    channel_images = [ndi.interpolation.affine_transform(x_channel, final_affine_matrix,
                      final_offset, order=0, mode='nearest') for x_channel in x]
    
    x = np.stack(channel_images, axis=0)
    x = np.rollaxis(x, 0, 3)
    
    return x

def random_transform(x, rg,height_shift_range,width_shift_range,shear_range,zoom_range):
    
    h, w = x.shape[0], x.shape[1]
    
    theta = np.pi / 180 * np.random.uniform(-rg, rg)
    rotation_matrix = np.array([[np.cos(theta), -np.sin(theta), 0],
                                [np.sin(theta), np.cos(theta), 0],
                                [0, 0, 1]])
    
    tx = h * np.random.uniform(-height_shift_range, height_shift_range) 
    ty = w * np.random.uniform(-width_shift_range, width_shift_range) 
    translation_matrix = np.array([[1, 0, tx],[0, 1, ty],[0, 0, 1]])
    
    shear = np.random.uniform(-shear_range, shear_range) #0.5
    shear_matrix = np.array([[1, -np.sin(shear), 0],[0, np.cos(shear), 0],[0, 0, 1]])
    
    
    zx, zy = np.random.uniform(zoom_range[0], zoom_range[1], 2)
    zoom_matrix = np.array([[zx, 0, 0],[0, zy, 0],[0, 0, 1]])
        
        
    transform_matrix = np.dot(np.dot(np.dot(rotation_matrix, translation_matrix), shear_matrix), zoom_matrix) 
    transform_matrix = transform_matrix_offset_center(transform_matrix, h, w)
    
    x = apply_transform(x, transform_matrix)
    return x

def transform_matrix_offset_center(matrix, x, y):
    o_x = float(x) / 2 + 0.5
    o_y = float(y) / 2 + 0.5
    offset_matrix = np.array([[1, 0, o_x], [0, 1, o_y], [0, 0, 1]])
    reset_matrix = np.array([[1, 0, -o_x], [0, 1, -o_y], [0, 0, 1]])
    transform_matrix = np.dot(np.dot(offset_matrix, matrix), reset_matrix)
    return transform_matrix

def trans(X): 
    for element in itertools.cycle(X):
        yield random_transform(element, rg=0.2,height_shift_range=0.1,width_shift_range=0.1,shear_range=0.5,zoom_range=(0.9,1.1))

def forward(X):
    for element in itertools.cycle(X):
        yield element
        
Xtrain_iterator = trans(X_train)
ytrain_iterator = forward(y_train)



Xv_iterator = forward(X_v)
Xt_iterator = forward(X_t)
yv_iterator = forward(y_v)
yt_iterator = forward(y_t)

Question 2

Describe how you set up the training, validation and testing data for your model. If you generated additional data, why?

Answer:

I generated training data on the fly (setup a pipeline inspired by Keras' ImageDataGenerator (i also took their concept of using ndimage and matrix transforms instead of a real image library). This way the training loop does not see the exact same image (but instead the slightly jittered one). This was the biggest improvement i made to get to the 98% vs 95% I split the test data 50%/50% into validation and test

In [8]:
import tensorflow as tf
#batch_size = 256
patch_size = 3
depth = 10
num_hidden = 512


tf_data = tf.placeholder(tf.float32, shape=(None, 32, 32, 3))
tf_labels = tf.placeholder(tf.float32, shape=(None, n_classes))
tf_keep = tf.placeholder(tf.float32)


# Variables.
conv1_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, 3, 32], stddev=0.1))
#conv1_biases = tf.Variable(tf.zeros([32])) # I dont use these anymore


conv2_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, 32, 32], stddev=0.1))
#conv2_biases = tf.Variable(tf.zeros([32]))

conv3_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, 32, 64], stddev=0.1))
#conv3_biases = tf.Variable(tf.zeros([64]))


conv4_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, 64, 64], stddev=0.1))
#conv4_biases = tf.Variable(tf.zeros([64]))


conv5_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, 64, 128], stddev=0.1))
conv5_biases = tf.Variable(tf.zeros([128]))

conv6_weights = tf.Variable(tf.truncated_normal([patch_size, patch_size, 128, 128], stddev=0.1))
conv6_biases = tf.Variable(tf.zeros([128]))




layer3_weights = tf.Variable(tf.truncated_normal([2048, num_hidden], stddev=0.1))
layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
layer4_weights = tf.Variable(tf.truncated_normal([num_hidden, n_classes], stddev=0.1))
layer4_biases = tf.Variable(tf.constant(1.0, shape=[n_classes]))


def model(data):
    #cov
    data = tf.reshape(data, [-1,32,32,3])
    
    conv = tf.nn.conv2d(data, conv1_weights, [1, 1, 1, 1], padding='SAME')
    #conv = tf.nn.bias_add(conv, conv1_biases)
    conv = tf.nn.relu(conv)
    conv = tf.nn.conv2d(conv, conv2_weights, [1, 1, 1, 1], padding='SAME')
    #conv = tf.nn.bias_add(conv, conv2_biases)
    conv = tf.nn.max_pool(conv, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],padding='SAME')

   
    
    conv = tf.nn.conv2d(conv, conv3_weights, [1, 1, 1, 1], padding='SAME')
    #conv = tf.nn.bias_add(conv, conv3_biases)
    conv = tf.nn.relu(conv)
    conv = tf.nn.conv2d(conv, conv4_weights, [1, 1, 1, 1], padding='SAME')
    #conv = tf.nn.bias_add(conv, conv4_biases)
    conv = tf.nn.max_pool(conv, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],padding='SAME')
    
    conv = tf.nn.conv2d(conv, conv5_weights, [1, 1, 1, 1], padding='SAME')
    conv = tf.nn.relu(conv)
    conv = tf.nn.conv2d(conv, conv6_weights, [1, 1, 1, 1], padding='SAME')
    conv = tf.nn.max_pool(conv, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],padding='SAME')
    
    
    

    shape = conv.get_shape().as_list()
    reshape = tf.reshape(conv, [-1,2048])

    reshape = tf.nn.dropout(reshape, tf_keep)
    
    hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)
    hidden_drop = tf.nn.dropout(hidden, tf_keep)
    return tf.matmul(hidden_drop, layer4_weights) + layer4_biases
  

logits = model(tf_data)
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits, tf_labels))
prediction = tf.nn.softmax(logits)

optimizer = tf.train.AdamOptimizer(0.001).minimize(loss)

correct_prediction = tf.equal(tf.argmax(prediction,1), tf.argmax(tf_labels,1))
tf_accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))

Question 3

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.

Answer:

My model is a 6 layer CNN with 3x3 patches and dropout layers every two CNN layers, increasing the depth from 3 to 32 to 64 to 128 and finally a dropout layer before a dense layer with 512 nodes another dropout layer and the output. This seems very standard besides the fact that I do not use biases - with bias my val accuracies went down to smaller than 98%

I tried a lot of more architectures but this got the best result. I would be very interested I better designs though! Please comment on how to get the 99% mentioned in some papers i tried some of those architectures in the papers but did not get good results (about 95%)

In [9]:
batch_size = 256
epoch = 100
num_steps = int(np.ceil(n_train / batch_size))
num_steps_val = int(np.ceil(n_test / batch_size / 2))
num_steps_test = int(np.ceil(n_test / batch_size / 2))

def bar(i):
    return "".join((["["]+["="]*int(i/5+1) + [" "]*(20-int(i/5+1))+["]"])) + " %03d%%"%(i+1)


Xtrain_iterator = trans(X_train)
ytrain_iterator = forward(y_train)

Xv_iterator = forward(X_v)
Xt_iterator = forward(X_t)
yv_iterator = forward(y_v)
yt_iterator = forward(y_t)
In [10]:
with tf.Session() as session:
    tf.initialize_all_variables().run()
    for i in range(epoch):
        loss1 = []
        loss2 = []
        acc1 = []
        acc2 = []
        
        for j in range(num_steps):
            print("\r %d/%d"%(j,num_steps), end="")
            print(bar( int(j/num_steps*100) ), end="")
            
            batch_data = np.array([Xtrain_iterator.__next__() for i in range(batch_size)])
            batch_labels = np.array([ytrain_iterator.__next__() for i in range(batch_size)])
            feed_dict = {tf_data : batch_data, tf_labels : batch_labels, tf_keep:0.4}
            _, l, predictions,accuracy = session.run([optimizer, loss, prediction,tf_accuracy ], feed_dict=feed_dict)
            loss1.append(l)
            acc1.append(accuracy)
        for j in range(num_steps_val):
            
            batch_data = np.array([Xv_iterator.__next__() for i in range(batch_size)])
            batch_labels = np.array([yv_iterator.__next__() for i in range(batch_size)])
            feed_dict = {tf_data : batch_data, tf_labels : batch_labels, tf_keep:1.0}
            l, predictions,accuracy = session.run([loss, prediction,tf_accuracy ], feed_dict=feed_dict)
            loss2.append(l)
            acc2.append(accuracy)
        print("")
        print("Epoch %d/%d" % (i+1,epoch))
        print("Train Loss: %f Validation Loss: %f" % (np.mean(np.array(loss1)), np.mean(np.array(loss2))))
        print("Train Acc: %f Validation Acc: %f" % (np.mean(np.array(acc1)), np.mean(np.array(acc2))))

    
    saver = tf.train.Saver()
    save_path = saver.save(session, "./model-final.ckpt")
 153/154[====================] 100%
Epoch 1/100
Train Loss: 4.741857 Validation Loss: 2.765028
Train Acc: 0.103820 Validation Acc: 0.307187
 153/154[====================] 100%
Epoch 2/100
Train Loss: 2.603843 Validation Loss: 1.663922
Train Acc: 0.283152 Validation Acc: 0.506406
 153/154[====================] 100%
Epoch 3/100
Train Loss: 1.897590 Validation Loss: 1.269165
Train Acc: 0.424335 Validation Acc: 0.604531
 153/154[====================] 100%
Epoch 4/100
Train Loss: 1.461999 Validation Loss: 0.952641
Train Acc: 0.531072 Validation Acc: 0.685156
 153/154[====================] 100%
Epoch 5/100
Train Loss: 1.143989 Validation Loss: 0.722600
Train Acc: 0.623935 Validation Acc: 0.748438
 153/154[====================] 100%
Epoch 6/100
Train Loss: 0.908119 Validation Loss: 0.533661
Train Acc: 0.697469 Validation Acc: 0.815156
 153/154[====================] 100%
Epoch 7/100
Train Loss: 0.722559 Validation Loss: 0.445251
Train Acc: 0.754946 Validation Acc: 0.847344
 153/154[====================] 100%
Epoch 8/100
Train Loss: 0.592421 Validation Loss: 0.334512
Train Acc: 0.802531 Validation Acc: 0.885781
 153/154[====================] 100%
Epoch 9/100
Train Loss: 0.468629 Validation Loss: 0.253881
Train Acc: 0.843243 Validation Acc: 0.918594
 153/154[====================] 100%
Epoch 10/100
Train Loss: 0.386103 Validation Loss: 0.220801
Train Acc: 0.869775 Validation Acc: 0.930156
 153/154[====================] 100%
Epoch 11/100
Train Loss: 0.328117 Validation Loss: 0.222767
Train Acc: 0.891437 Validation Acc: 0.936719
 153/154[====================] 100%
Epoch 12/100
Train Loss: 0.287538 Validation Loss: 0.201173
Train Acc: 0.906529 Validation Acc: 0.943281
 153/154[====================] 100%
Epoch 13/100
Train Loss: 0.243666 Validation Loss: 0.171940
Train Acc: 0.919795 Validation Acc: 0.948281
 153/154[====================] 100%
Epoch 14/100
Train Loss: 0.207683 Validation Loss: 0.165856
Train Acc: 0.932554 Validation Acc: 0.956250
 153/154[====================] 100%
Epoch 15/100
Train Loss: 0.183053 Validation Loss: 0.157589
Train Acc: 0.940595 Validation Acc: 0.955000
 153/154[====================] 100%
Epoch 16/100
Train Loss: 0.164475 Validation Loss: 0.132346
Train Acc: 0.945896 Validation Acc: 0.963125
 153/154[====================] 100%
Epoch 17/100
Train Loss: 0.150463 Validation Loss: 0.166427
Train Acc: 0.951045 Validation Acc: 0.955312
 153/154[====================] 100%
Epoch 18/100
Train Loss: 0.142144 Validation Loss: 0.120104
Train Acc: 0.954596 Validation Acc: 0.963906
 153/154[====================] 100%
Epoch 19/100
Train Loss: 0.140661 Validation Loss: 0.112570
Train Acc: 0.955738 Validation Acc: 0.967031
 153/154[====================] 100%
Epoch 20/100
Train Loss: 0.119765 Validation Loss: 0.106109
Train Acc: 0.961572 Validation Acc: 0.970312
 153/154[====================] 100%
Epoch 21/100
Train Loss: 0.105180 Validation Loss: 0.095512
Train Acc: 0.966848 Validation Acc: 0.970781
 153/154[====================] 100%
Epoch 22/100
Train Loss: 0.102068 Validation Loss: 0.101949
Train Acc: 0.967482 Validation Acc: 0.969688
 153/154[====================] 100%
Epoch 23/100
Train Loss: 0.107105 Validation Loss: 0.111844
Train Acc: 0.966873 Validation Acc: 0.961250
 153/154[====================] 100%
Epoch 24/100
Train Loss: 0.089405 Validation Loss: 0.134450
Train Acc: 0.972428 Validation Acc: 0.965625
 153/154[====================] 100%
Epoch 25/100
Train Loss: 0.089139 Validation Loss: 0.082011
Train Acc: 0.972555 Validation Acc: 0.975000
 153/154[====================] 100%
Epoch 26/100
Train Loss: 0.088224 Validation Loss: 0.086798
Train Acc: 0.971794 Validation Acc: 0.975781
 153/154[====================] 100%
Epoch 27/100
Train Loss: 0.086397 Validation Loss: 0.123766
Train Acc: 0.972428 Validation Acc: 0.970937
 153/154[====================] 100%
Epoch 28/100
Train Loss: 0.076772 Validation Loss: 0.091140
Train Acc: 0.975751 Validation Acc: 0.975625
 153/154[====================] 100%
Epoch 29/100
Train Loss: 0.072609 Validation Loss: 0.087518
Train Acc: 0.977374 Validation Acc: 0.977500
 153/154[====================] 100%
Epoch 30/100
Train Loss: 0.077557 Validation Loss: 0.117537
Train Acc: 0.976258 Validation Acc: 0.972812
 153/154[====================] 100%
Epoch 31/100
Train Loss: 0.075655 Validation Loss: 0.110637
Train Acc: 0.976639 Validation Acc: 0.972031
 153/154[====================] 100%
Epoch 32/100
Train Loss: 0.068233 Validation Loss: 0.103978
Train Acc: 0.979606 Validation Acc: 0.973437
 153/154[====================] 100%
Epoch 33/100
Train Loss: 0.071518 Validation Loss: 0.133590
Train Acc: 0.977755 Validation Acc: 0.967344
 153/154[====================] 100%
Epoch 34/100
Train Loss: 0.065018 Validation Loss: 0.106051
Train Acc: 0.980164 Validation Acc: 0.972656
 153/154[====================] 100%
Epoch 35/100
Train Loss: 0.066357 Validation Loss: 0.094675
Train Acc: 0.980012 Validation Acc: 0.982031
 153/154[====================] 100%
Epoch 36/100
Train Loss: 0.067602 Validation Loss: 0.117024
Train Acc: 0.979759 Validation Acc: 0.977969
 153/154[====================] 100%
Epoch 37/100
Train Loss: 0.067860 Validation Loss: 0.092799
Train Acc: 0.979429 Validation Acc: 0.978594
 153/154[====================] 100%
Epoch 38/100
Train Loss: 0.059923 Validation Loss: 0.080542
Train Acc: 0.982194 Validation Acc: 0.981562
 153/154[====================] 100%
Epoch 39/100
Train Loss: 0.057248 Validation Loss: 0.094655
Train Acc: 0.983360 Validation Acc: 0.975781
 153/154[====================] 100%
Epoch 40/100
Train Loss: 0.054631 Validation Loss: 0.094851
Train Acc: 0.984223 Validation Acc: 0.977188
 153/154[====================] 100%
Epoch 41/100
Train Loss: 0.056434 Validation Loss: 0.096709
Train Acc: 0.982752 Validation Acc: 0.975781
 153/154[====================] 100%
Epoch 42/100
Train Loss: 0.056551 Validation Loss: 0.099356
Train Acc: 0.983259 Validation Acc: 0.977344
 153/154[====================] 100%
Epoch 43/100
Train Loss: 0.047829 Validation Loss: 0.136616
Train Acc: 0.985212 Validation Acc: 0.972188
 153/154[====================] 100%
Epoch 44/100
Train Loss: 0.058153 Validation Loss: 0.095343
Train Acc: 0.983969 Validation Acc: 0.977031
 153/154[====================] 100%
Epoch 45/100
Train Loss: 0.052538 Validation Loss: 0.091416
Train Acc: 0.984857 Validation Acc: 0.977656
 153/154[====================] 100%
Epoch 46/100
Train Loss: 0.055910 Validation Loss: 0.093536
Train Acc: 0.983918 Validation Acc: 0.977969
 153/154[====================] 100%
Epoch 47/100
Train Loss: 0.051532 Validation Loss: 0.098518
Train Acc: 0.985035 Validation Acc: 0.979062
 153/154[====================] 100%
Epoch 48/100
Train Loss: 0.049721 Validation Loss: 0.082475
Train Acc: 0.984654 Validation Acc: 0.982031
 153/154[====================] 100%
Epoch 49/100
Train Loss: 0.051369 Validation Loss: 0.131771
Train Acc: 0.985440 Validation Acc: 0.968906
 153/154[====================] 100%
Epoch 50/100
Train Loss: 0.049843 Validation Loss: 0.080713
Train Acc: 0.986074 Validation Acc: 0.980000
 153/154[====================] 100%
Epoch 51/100
Train Loss: 0.042288 Validation Loss: 0.105079
Train Acc: 0.987647 Validation Acc: 0.977031
 153/154[====================] 100%
Epoch 52/100
Train Loss: 0.044059 Validation Loss: 0.101542
Train Acc: 0.987393 Validation Acc: 0.976094
 153/154[====================] 100%
Epoch 53/100
Train Loss: 0.045836 Validation Loss: 0.093077
Train Acc: 0.986506 Validation Acc: 0.982188
 153/154[====================] 100%
Epoch 54/100
Train Loss: 0.048713 Validation Loss: 0.115535
Train Acc: 0.986455 Validation Acc: 0.973750
 153/154[====================] 100%
Epoch 55/100
Train Loss: 0.046246 Validation Loss: 0.095480
Train Acc: 0.986582 Validation Acc: 0.980937
 153/154[====================] 100%
Epoch 56/100
Train Loss: 0.042460 Validation Loss: 0.076321
Train Acc: 0.988408 Validation Acc: 0.984219
 153/154[====================] 100%
Epoch 57/100
Train Loss: 0.044485 Validation Loss: 0.093350
Train Acc: 0.987520 Validation Acc: 0.977656
 153/154[====================] 100%
Epoch 58/100
Train Loss: 0.048840 Validation Loss: 0.090742
Train Acc: 0.986303 Validation Acc: 0.982656
 153/154[====================] 100%
Epoch 59/100
Train Loss: 0.043223 Validation Loss: 0.103201
Train Acc: 0.988154 Validation Acc: 0.977969
 153/154[====================] 100%
Epoch 60/100
Train Loss: 0.043739 Validation Loss: 0.090846
Train Acc: 0.987520 Validation Acc: 0.980625
 153/154[====================] 100%
Epoch 61/100
Train Loss: 0.040112 Validation Loss: 0.138408
Train Acc: 0.988687 Validation Acc: 0.977656
 153/154[====================] 100%
Epoch 62/100
Train Loss: 0.047056 Validation Loss: 0.130033
Train Acc: 0.987495 Validation Acc: 0.979375
 153/154[====================] 100%
Epoch 63/100
Train Loss: 0.047038 Validation Loss: 0.153742
Train Acc: 0.987444 Validation Acc: 0.970469
 153/154[====================] 100%
Epoch 64/100
Train Loss: 0.046434 Validation Loss: 0.101411
Train Acc: 0.987343 Validation Acc: 0.981406
 153/154[====================] 100%
Epoch 65/100
Train Loss: 0.040495 Validation Loss: 0.100505
Train Acc: 0.988865 Validation Acc: 0.979219
 153/154[====================] 100%
Epoch 66/100
Train Loss: 0.036317 Validation Loss: 0.064117
Train Acc: 0.989524 Validation Acc: 0.985469
 153/154[====================] 100%
Epoch 67/100
Train Loss: 0.036116 Validation Loss: 0.082577
Train Acc: 0.990260 Validation Acc: 0.984062
 153/154[====================] 100%
Epoch 68/100
Train Loss: 0.042671 Validation Loss: 0.098285
Train Acc: 0.988510 Validation Acc: 0.976250
 153/154[====================] 100%
Epoch 69/100
Train Loss: 0.043296 Validation Loss: 0.067114
Train Acc: 0.988484 Validation Acc: 0.985781
 153/154[====================] 100%
Epoch 70/100
Train Loss: 0.033848 Validation Loss: 0.080980
Train Acc: 0.989372 Validation Acc: 0.983594
 153/154[====================] 100%
Epoch 71/100
Train Loss: 0.040084 Validation Loss: 0.103867
Train Acc: 0.988966 Validation Acc: 0.977344
 153/154[====================] 100%
Epoch 72/100
Train Loss: 0.039087 Validation Loss: 0.071539
Train Acc: 0.989524 Validation Acc: 0.981875
 153/154[====================] 100%
Epoch 73/100
Train Loss: 0.036969 Validation Loss: 0.089239
Train Acc: 0.989524 Validation Acc: 0.981875
 153/154[====================] 100%
Epoch 74/100
Train Loss: 0.038268 Validation Loss: 0.100248
Train Acc: 0.990082 Validation Acc: 0.981562
 153/154[====================] 100%
Epoch 75/100
Train Loss: 0.048070 Validation Loss: 0.077461
Train Acc: 0.988332 Validation Acc: 0.981562
 153/154[====================] 100%
Epoch 76/100
Train Loss: 0.039102 Validation Loss: 0.095486
Train Acc: 0.989068 Validation Acc: 0.982969
 153/154[====================] 100%
Epoch 77/100
Train Loss: 0.036440 Validation Loss: 0.100865
Train Acc: 0.989981 Validation Acc: 0.980937
 153/154[====================] 100%
Epoch 78/100
Train Loss: 0.040363 Validation Loss: 0.063962
Train Acc: 0.989930 Validation Acc: 0.985625
 153/154[====================] 100%
Epoch 79/100
Train Loss: 0.035761 Validation Loss: 0.088774
Train Acc: 0.990995 Validation Acc: 0.981094
 153/154[====================] 100%
Epoch 80/100
Train Loss: 0.035579 Validation Loss: 0.082785
Train Acc: 0.990666 Validation Acc: 0.984688
 153/154[====================] 100%
Epoch 81/100
Train Loss: 0.039332 Validation Loss: 0.082351
Train Acc: 0.990031 Validation Acc: 0.982813
 153/154[====================] 100%
Epoch 82/100
Train Loss: 0.038238 Validation Loss: 0.099046
Train Acc: 0.989423 Validation Acc: 0.980000
 153/154[====================] 100%
Epoch 83/100
Train Loss: 0.039699 Validation Loss: 0.074228
Train Acc: 0.990184 Validation Acc: 0.984844
 153/154[====================] 100%
Epoch 84/100
Train Loss: 0.039309 Validation Loss: 0.066768
Train Acc: 0.989347 Validation Acc: 0.983281
 153/154[====================] 100%
Epoch 85/100
Train Loss: 0.031317 Validation Loss: 0.087538
Train Acc: 0.991908 Validation Acc: 0.982344
 153/154[====================] 100%
Epoch 86/100
Train Loss: 0.033778 Validation Loss: 0.091466
Train Acc: 0.990792 Validation Acc: 0.985000
 153/154[====================] 100%
Epoch 87/100
Train Loss: 0.035202 Validation Loss: 0.066306
Train Acc: 0.990589 Validation Acc: 0.984688
 153/154[====================] 100%
Epoch 88/100
Train Loss: 0.029881 Validation Loss: 0.080457
Train Acc: 0.991680 Validation Acc: 0.982500
 153/154[====================] 100%
Epoch 89/100
Train Loss: 0.030955 Validation Loss: 0.068991
Train Acc: 0.991731 Validation Acc: 0.985000
 153/154[====================] 100%
Epoch 90/100
Train Loss: 0.028338 Validation Loss: 0.073279
Train Acc: 0.991934 Validation Acc: 0.984219
 153/154[====================] 100%
Epoch 91/100
Train Loss: 0.037400 Validation Loss: 0.065598
Train Acc: 0.990615 Validation Acc: 0.986094
 153/154[====================] 100%
Epoch 92/100
Train Loss: 0.034741 Validation Loss: 0.076880
Train Acc: 0.991604 Validation Acc: 0.981250
 153/154[====================] 100%
Epoch 93/100
Train Loss: 0.031684 Validation Loss: 0.067717
Train Acc: 0.992010 Validation Acc: 0.985781
 153/154[====================] 100%
Epoch 94/100
Train Loss: 0.027004 Validation Loss: 0.051603
Train Acc: 0.992746 Validation Acc: 0.989219
 153/154[====================] 100%
Epoch 95/100
Train Loss: 0.038652 Validation Loss: 0.081392
Train Acc: 0.990234 Validation Acc: 0.984844
 153/154[====================] 100%
Epoch 96/100
Train Loss: 0.036613 Validation Loss: 0.070774
Train Acc: 0.990336 Validation Acc: 0.980937
 153/154[====================] 100%
Epoch 97/100
Train Loss: 0.034667 Validation Loss: 0.057130
Train Acc: 0.991148 Validation Acc: 0.987969
 153/154[====================] 100%
Epoch 98/100
Train Loss: 0.031989 Validation Loss: 0.083297
Train Acc: 0.991553 Validation Acc: 0.984375
 153/154[====================] 100%
Epoch 99/100
Train Loss: 0.038759 Validation Loss: 0.073155
Train Acc: 0.990031 Validation Acc: 0.983281
 153/154[====================] 100%
Epoch 100/100
Train Loss: 0.034430 Validation Loss: 0.079768
Train Acc: 0.991198 Validation Acc: 0.986250

Typical result after 100 Epoch:

  • Train Loss: 0.034430 Validation Loss: 0.079768
  • Train Acc: 0.991198 Validation Acc: 0.986250

Lets have a look at the test data next

In [11]:
loss_array = []
acc_array = []
f = []
p = []

with tf.Session() as sess:
    saver = tf.train.Saver()
    saver.restore(sess, "./model-final.ckpt")
    print("Model restored.")
    
    for i in range(np.ceil(len(X_t)/1000).astype(np.int)):
        batch_data = X_t[i*1000:i*1000+1000]
        batch_labels = y_t[i*1000:i*1000+1000]
        feed_dict = {tf_data : batch_data, tf_labels : batch_labels, tf_keep:1.0}
        l2, predictions,a = sess.run([loss, prediction,tf_accuracy], feed_dict=feed_dict)
        loss_array.append(l2)
        acc_array.append(a)
        
        f = np.array(np.concatenate([f, np.argmax(predictions, 1) != np.argmax(batch_labels, 1)]))
        p = np.array(np.concatenate([p, np.argmax(predictions, 1)]))

print("Accuracy %")
print(np.mean(acc_array))
print("Loss")
print(np.mean(loss_array))
print("")
Model restored.
Accuracy %
0.984968
Loss
0.119458

In [12]:
f = f.astype(np.bool)
print("Overview of failiures")
draw(X_t[f], np.argmax(y_t[f], axis=1))
print("Predictions")
draw(X_t[f], p[f].astype(np.int))
Overview of failiures
Predictions

Question 4

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:

I used

  • Adam optimizer (seems to be the de-facto standard)
  • 256 batch
  • 100 epochs (not exact it leaks a bit because the last set is not quite 256 so i start over again early)
  • dropout rate of 50%

These parameters are not rigorously optimized but more a this works approach

Question 5

What approach did you take in coming up with a solution to this problem?

Answer:

Since it is an image recognition problem I started with a basic CNN (like the mnist one in the tensorflow tutorial on the tensorflow website). Then increased the size and added more dropout but that limited me to 94% or so so i added image jittering.

I also tried the techniques used in the Sermanet Paper but was not successful with those at all.


Step 3: Test a Model on New Images

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

In [13]:
import os
import skimage.data
from skimage.transform import resize

newdata = np.array([resize(skimage.data.imread("./collect-japan/"+name), 
                  output_shape=(32,32,3)) for name in os.listdir("./collect-japan/")])

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [14]:
newdata = equalize_hist(newdata)

top5 = tf.nn.top_k(prediction, 5)
with tf.Session() as sess:
    saver = tf.train.Saver()
    saver.restore(sess, "./model-final.ckpt")
    print("Model restored.")
    for i in range(1):
        
        feed_dict = {tf_data : newdata, tf_keep:1.0}
        predictions,t5 = sess.run([prediction, top5], feed_dict=feed_dict)
Model restored.

Question 6

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It would be helpful to plot the images in the notebook.

Answer:

I took a few more... one challenge is that they are Japanese (local) and that some are not in the set (no parking) so perfect results cannot be expected but i want to see what the network says anyway. Lets have a look:

In [15]:
draw(newdata, t5[1][:,0])

Question 7

Is your model able to perform equally well on captured pictures or a live camera stream when compared to testing on the dataset?

Answer:

Yes it did really well! It can even classify a Japanese Stop even though it has never seen that although that is more luck.

Its not able to classify the no parking correctly. But that because it surprisingly not in the train dataset... looking at the table below it is also quite confident in the wrong prediction. Not good but can only be fixed with more data. It confuses it with Keep right which is indeed somewhat similar and in bad lighting with speed limit and No vechiles (but those only with low certainty.

Other problems are also caused by the sign not being in the train set and therefore it cannot find anything... No trucks is seen as 99% Speed limit (100km/h).

In [ ]:
import matplotlib.pyplot as plt
short = [s[:30] for s in df.as_matrix()[:,1]]
for loop in range(3):
    figure(figsize=(20,3))
    for signnum in range(5):
        
        subplot(151+signnum)  
        title("Image %d" % (loop*5+signnum+1))
        rects1 = plt.bar(np.arange(5),t5[0][loop*5+signnum], 0.5)
        s = [s[:30] for s in df.iloc[t5[1][loop*5+signnum].astype(np.int)]['SignName']]
        plt.xticks(np.arange(5)+0.25, s, rotation='vertical')
        xlim(0,5)
        ylim(0,1)
    show()
        

Question 8

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

Answer: See above for visualization.

  • Sign 6-9, 13, 15 are uncertain
  • In general it seems that low lightning Images are more uncertain. There are some images that the network is very certain about (like Image 14 (Keep right)) but are still wrong. Those are usually not german signs or signs like no parking which are not in the training set at all.
  • Image 7 and 9 are interesting because it classifies incorrectly but not confidently (for both the correct result is on rank 3)

More data in bad lightning might help to improve performance in bad lightning (I also tried preprocessing with equalize_adapthist which improves bad lightning performance but increases sensitivity to noise.). Against unknown signs only training with these signs will help.

Question 9

If necessary, provide documentation for how an interface was built for your model to load and classify newly-acquired images.

Answer: I only took some pictures with my phone and selected the part with the sign manually. Then I load and resize the images and then they run through the same queue as the dataset.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.